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Capsicum Update 2019 

Every day the FreeBSD community makes improvements, and in 
this article, we take a look at how Capsicum has changed over the 
past year. By Mariusz Zaborski 

Improving Memory Permissions 

in FreeBSD This article covers the basics of machine-inde¬ 
pendent memory permissions that fit the current mmap() permission 
model. By Brooks Davis 

Configuring Full-disk Encryption 

on FreeBSD While there are multiple ways to configure full- 
disc encryption on FreeBSD, this article focuses on one method which 
provides an easy route to follow and get started using GEU. 

By Roller Angel 


Intorviow Excerpts from Kris Moore and Allan Jude's very 

interesting 2014 interview with Pawel Dawidek called, Gift from the 
Sun, from BSD Now 62. 
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Foundation Letter By John Baldwin. 

We Get Letters! Enjoy Being Doomed. By Michael W Lucas. 

Conference Report(s) 

• COSCUP-Taipei is definitely a place where we should strongly promote the BSDs as the crowd 

was very interested in what the OS can provide. By Benedict Reuschling. 

• vBSDcon-l came away from this enjoyable conference with deeper knowledge of the subjects 

covered and a stronger connection with the community. By Brad Alexander. 

• EuroBSDcon-This conference in Norway was wonderful and gave me a lot of topics to 

research and possible project ideas to explore. By Chris Iroanya. 

Events Calendar By Anne Dickison. 
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Donate to the Foundation! 

You already know that FreeBSD is an internationally 
recognized leader in providing a high-performance, 
secure, and stable operating system. It's because of 
you. Your donations have a direct impact on the Project. 

Please consider making a gift to support FreeBSD for the 
coming year. It's only with your help that we can continue 
and increase our support to make FreeBSD the high- 
performance, secure, and reliable OS you know and love! 

Your investment will help: 

• Funding Projects to Advance FreeBSD 

• Increasing Our FreeBSD Advocacy and 
Marketing Efforts 

• Providing Additional Conference 
Resources and Travel Grants 

• Continued Development of the FreeBSD 
Journal 

• Protecting FreeBSD IP and Providing 
Legal Support to the Project 

• Purchasing Hardware to Build and 
Improve FreeBSD Project Infrastructure 
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LETTER 

from the Board 


W elcome to the September/October issue of 
the FreeBSD Journal. These past few months 
have seen a flurry of activity, including three 
conferences and the final stages of the 12.1 
release process. Developers and users met in northern 
Virginia near Washington, DC, forvBSDCon 2019 in early 
September, followed by EuroBSDCon 2019 in Lillehammer 
two weeks later. Developers met again in Santa Clara, 
California, for a developer and vendor summit in mid- 
October. As Brad Alexander relates, conferences are an 
excellent place for FreeBSD community members to connect 
with each other. Bugs previously open for months can be 
resolved within a few hours at a conference due to the 
increased productivity of face-to-face time. FreeBSD Journal 
authors and editorial board members attend many of these 
conferences, and we would be happy to meet you in per¬ 
son. You can find lists of upcoming events on the FreeBSD 
events page at h ttps://www.free bsd. org/events/event s.html 
as well as the FreeBSD Foundation events page at 
h ttps://www.fre ebsdfoundation.org /news-an d- 

events/upcoming-events/ . 

One of the things I most like about FreeBSD is the com¬ 
munity of users and developers. It is true that FreeBSD pro¬ 
duces source code, documentation, packages, and releases. 
However, those products are generated and used by individ¬ 
uals. We spend time not only working together, but also 
building friendships as we chat on IRC or Slack, hack on 
code at conferences, or enjoy a nice dinner or museum visit 
during a conference. I have several close friends in FreeBSD's 
community, and as a community we share life together. 

This includes celebrating new faces in the community, new 
jobs and better jobs, and new offspring. Sharing life also 
entails supporting each other in hard times and mourning 
as we celebrate the lives of friends who have passed. This 
October, developer Kurt Lidl (lidl@) passed after a battle 
with cancer. We are grateful for all of Kurt's contributions 
to BSD ( https://www.f reeb sdfoundation.org/blog /i n-mem o- 
ry-of-kurt-lidl/). Our thoughts and prayers are with his family 
and friends. 

We always love to hear from our readers. If you have 
feedback on any of our articles or suggestions for topics for 
a future article, please email us at: 

info@freebsdjournal.conn 

John Baldwin 

Chair of the FreeBSD Journal Editorial Board 
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Capsicum 

UpddtG 2019 by Mariusz Zaborski 


FreeBSD is a general-purpose operating system. One 
of its goals is to provide a secure environment for its 
users. To meet that goal, among others, FreeBSD has 
introduced a Capsicum framework. Every day, the 
FreeBSD community works to make improvements, 
and in this article, we will take a look at how 
Capsicum has changed over the past year. 

T he Capsicum framework provides tight isolation between processes. 
When a process enters the capability mode (sandbox), it doesn't have 
access to any global namespace. The process can enter this state by 
using cap_enter(1) syscall. The difference between Capsicum and 
other popular sandbox frameworks is that Capsicum focuses on the capabili¬ 
ties of processes instead of filtering syscalls. In Capsicum, descriptors repre¬ 
sent capabilities. The descriptors are perfect for capabilities since we can 
duplicate ( dup(2)), send them to other processes (through Unix Domain 
Socket), or revoke them (dose(2)). 

Another component of the framework is capability rights. These allow us 
to limit capabilities (file descriptors) even further. The descriptor can be limit¬ 
ed to be readable {CAP_READ) or writable ( CAP_WRITE ) using a 
cap_rights_limit(2) syscall. The capabilities can be limited even if they can 
reposition their offset in the file ( CAP_SEEK ). Capsicum allows the purpose 
of the descriptor to be defined very precisely. There are over 50 capabilities 
right now. 

For a more detailed description of the framework, we recommend you 
look at other issues of the FreeBSD Journal where you will find more 
detailed descriptions of Capsicum [1] [2], 


Casper 

Other primitives necessary to understand the Capsicum environment are 
Casper and Casper services. Casper provides functionality not available in 
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capability mode through convenient APIs, making Capsicum more practical. 
Casper accomplishes this by utilizing the privileged separation of the process. 
After creating a new Casper instance, the initial process forks. Next, the unprivi¬ 
leged process enters capability mode. If the sandboxed application wants to 
perform some forbidden operation in Capsicum, it has to delegate this work to 
Casper. Casper communicates with a sandboxed application using a straightfor¬ 
ward API called libnv (or nvlist). 

Let's assume that an application wants to resolve the DNS name. Before 
entering capability mode, it can create Casper services— system.dns. Each time it 
needs to resolve a DNS, instead of calling the getaddrinfo function, it calls the 
cap_getaddrinfo. The Casper function has precisely the same API as standard 
libc, except it has one more argument that takes Casper connection. 

The Casper services API is straightforward. Thanks to this, the sandboxing 
application is much more comfortable. Without Casper, all applications develop¬ 
ers would need to reimplement the privileged separation routines to sandbox 
new application. 


Introducing fileargs 

The fileargs service allows programs to access a filesystem. This service makes it 
relatively easy to sandbox applications. The wc(1) and head(1) are examples of 
applications which depend on it. The fileargs service doesn't provide a full 
filesystems service. Its primary focus is to address the applications which, as an 
argument, takes a vector of files to open. Nevertheless, this service may also be 
used for different scenarios. 

The service is essential for two reasons. First, as described above, it allows a 
new applications to be sandboxed. Secondly, because this is the first service in 
which the API doesn't reflect the libc functions. 

Currently, the fileargs services provides two main functions—it allows the 
opening of files and to get their status. The filargs_open/fileargs_fopen func¬ 
tions allow the opening of a file from a given path and the fileargsjfstat func¬ 
tion provides the capability to gather the status. The primary function is 
fileargsjnit. This function initializes the Casper service. 


fileargs_t *fileargs_init(int argc, char *argv[], 
mode, cap_rights_t *rightsp, int operations); 


int flags, 


mode t 


The argc and argv arguments are just vectors within the files that applications 
should be able to open. The flags and mode argument aren't any different from 
an argument to open. Those arguments describe how the file should be opened 
and in which mode it should be created. Next is the list of capabilities that the 
newly opened file should keep. The last argument is which operations in service 
are permitted. For now, services define two operations, open ( FA_OPEN ) and 
Istat (IFA_L5TAT ). 

The fileargs_cinit function is very similar to the filargsjnit function. The only 
difference is that fileargs_cinit reuses already existing Casper instances. In the 
case of a filargsjnit function, the Casper service creates new instances. 
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@@ -79,6 +85,8 @@ main(int argc, char *argv[ ]) 
char *ep; 
off_t bytecnt; 

int ch, first, linecnt, eval; 

+ fileargs_t *fa; 

+ caprightst rights; 

linecnt = -1; 
eval = 0; 

@@ -106,13 +114,22 @@ main(int argc, char *argv[ ]) 
argc -= optind; 
argv += optind; 


+ fa = fileargs_init(argc, argv, ORDONLY, 0, 

+ cap_rights_init ( srights, CAPREAD, CAP_FSTAT, 

CAP_FCNTL )) ; 

+ if (fa == NULL) 

+ errx(l, "unable to init casper"); 

+ 

+ caph_cache_catpages () ; 

+ if (caph_limit_stdio( ) < 0 || caph_enter_casper( ) < 0) 

+ err(l, "unable to enter capability mode"); 

+ 


+ 

NULL) 


if (linecnt != -1 && bytecnt != -I) 11 

errx(l, "can't combine line and byte counts"); 
if (linecnt == -1) 

linecnt = 10; 
if ( *argv 1= NULL) { 

for (first = 1; *argv 1= NULL; ++argv) { 

if (( fp = fopen(*argv, "r")) == NULL) { 
if ((fp = fileargs_fopen(fa, *argv, "r")) == 

warn("%s", *argv); 
eval = 1; 

continue; 


Listing 1. The patch for sandboxing head(1). 



Listing 1 presents a patch for sandboxing the head(1). The patch is straightfor¬ 
ward. All we needed to do was initialize the Casper service with the right capa¬ 
bilities, pass argv and argc to it, and change the open function to the Casper 
version. Finally, we entered the capability mode. 

It is worth noting that the fileargs service API is still considered to be experi¬ 
mental and may change. 


Improving cap_sysctl 

The cap_sysctl services allow us to interact with the kernel state. In the original 
implementation, we introduced the cap._sysctlbyname function. However, when 
the sandboxing process of rtsol(8) and rtsold(8) began, it became clear it was not 
enough. The sysctl can be referred to in two ways—by its text representation and 
by its numeric representation. 

The process of sandboxing those applications motivates developers to extend 
cap_sysctl. The cap_sysctl and cap_sysctlnametomib functions were introduced. 
The first allows the manipulation of values through numeric representation. The 
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second makes it possible to fetch a numeric representation of Management 
Information Base (MIB) of sysctl from a given string representation. The interface 
of those two functions is very similar to their predecessor. The only change is that 
the Casper functions expect additional argument with the connection to the 
Casper service. 

The interface extension also meant that the limitation functions should be 
reworked. We introduced a new interface for the cap_sysctl service: 

• cap_sysctl_limit_init - initializes the limitation structures 

• cap_sysctl_limit_name - allows the limiting of single MIBs through name 
representation 

• cap_sysctl_limit_mib - allows the limiting of single MIBs through numeric 
representation 

• cap_sysctl_limit - sets the limits on given Casper service instances and frees 
all underlying structures. 

In Listing 2, we have an example of using it. First, we create a Casper instance 
with capjnit, and Casper service with cap_service_open, which is the standard 



cap_channel_t *capcas, *capsysctl; 
const char ‘name = "kern.trap_enotcap"; 
cap_sysctl_limit_t * limit; 
int value; 
size_t size; 

/* Open capability to Casper. */ 
capcas = cap_init(); 
if (capcas == NULL) 

err(l, "Unable to contact Casper"); 

/* Use Casper capability to create capability to the system.sysctl 
service. */ 

capsysctl = cap_service_open(capcas, "system.sysctl" ) ; 
if (capsysctl == NULL) 

err(l, "Unable to open system.sysctl service"); 

/* Close Casper capability, we don't need it anymore. */ 
capclose ( capcas ) ; 

/* Create limit for one MIB with read access only. */ 
limit = cap_sysctl_limit_init ( capsysctl ) ; 

( void )cap_sysctl_limit_name(limit, name, CAP_SYSCTL_READ) ; 

/* Limit system.sysctl. */ 
if (cap_sysctl_limit ( limit) < 0) 

err(l, "Unable to set limits"); 

/* Fetch value. */ 

if (cap_sysctlbyname ( capsysctl, name, svalue, &size, NULL, 0) < 0) 
err(l, "Unable to get value of sysctl"); 

printf("The value of %s is %d.\n", name, value); 


Listing 2. The main page example of usage cap_sysctl limits. 
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method. Next, we initialize sysctl limits. We limit our service only to one 
sysctl-Zcem. trap_enotcap. We can refer to it only with a text representation. 
The CAP_SYSCTL_READ also means that an application can only fetch the 
value of this sysctl. At the end of Listing 2, the program fetches that value. 

Private Services 

Mark Johnston did some more exciting work. When he was sandboxing 
rtsol(8) and rtsold(8), he implemented a private Casper service dedicated only 
to those two applications. The rtsold(8) is a daemon program to send ICMPv6 
Router Solicitation messages on the specified interfaces. The service is appli¬ 
cation specific, so there was no reason to make it publicly available. This 
approach may get us to the point where some services will be installed from 
the ports/packaging systems. His work allows us to see that the Casper serv¬ 
ice may also be used in different environments for process separation. 

The rtsol(8) and rtsold(8) used Casper to create a service for sending Router 
Solicitation messages on a raw ICMPv6 socket. This is accomplished by the 
cap_sendmsg service. Another private service, cap_script, is used to spawn 
and collect the status of scripts required by the rtsold daemon. The third and 
last service implemented for this program is capjlflags. This service is respon¬ 
sible for fetching the flags for the link-local IPv6 address on the specified 
interface. 

The rtsold(8) is an example of a sandboxed program within the Casper 
service that didn't require the implementing of general wild services. 

The Super Capsicumizer 9000 

Concealed behind this funny name is a small diamond. The Super 
Capsicumizer 9000, or just Capsicumizer, is an open-source project that has 
tried, with success, to implement the sandbox launcher that uses Capsicum. 
[3] AppArmor inspires the Capsicumizer. AppArmor is a mandatory access 
control system that allows process access to be limited. Its confinement is 
provided via profiles loaded into the kernel, typically on boot. The profiles can 
be managed by the administrator of the system and describe which resources 
the application should have access to. 

Capsicumizer is based on the profiles as well. The difference is that instead 
of loading profiles to the kernel, we run Capsicumizer in userland and allow 
Capsicum to handle limitations of the process. The patterns are defined using 
UCL syntax. 

The Capsicumizer uses a libpreopen library to all the resource pre-opened 
directory descriptors and using them in capability-safe libc wrappers. Thanks 
to that library, our application will have all the capabilities it requires. 

Currently, the limitation of Capsicumizer is that it allows only limited 
resources to the filesystems one. Unfortunately, defining or preconfiguring 
network access is not supported. 

The Capsicumizer is an exciting project that already allows us to sandbox 
applications without modifying them. For now, it is limited only to the filesys¬ 
tem. It would be interesting to see it combined with the Casper. With such a 
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Listing 3. An example of configuation from Capsicumizer 3000 which allows to sandbox gedit. 


combination, we would be able to sandbox a lot of applications without chang¬ 
ing their code. 

Summary 

The FreeBSD Capsicum framework is still under development but is already wide¬ 
ly used. The improvements to the Casper services, especially cap_fileargs, opens 
a whole new set of applications that can be easily sandboxed. Projects like 
Capsicumizer can get us to the point where administrators wanting to separate 
single process will not need to touch the code to achieve their aim. • 


BIBLIOGRAPHY 
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Improvlni Memory 


Permis 


in FraalSD 


by Brooks Davis 


The virtual address space of a process contains a number of 
physical pages mapped into memory. These might be pages 
from a program, a library, an ordinary file, or anonymous 
pages that begin life as a zeroed page. These mappings are 
maintained in a translation lookaside buffer (TLB). On modern 
architectures, the TLB allows pages to be mapped with a combina¬ 
tion of read, write, and execute permissions. This enables things 
like read-only sharing of code and data between processes for 
physical memory utilization. 


O lder architectures (e.g., MIPS, early i386) only supported read and write 
f permission, but modern CPUs generally support an execute permission 
as well. Used correctly, the execute permission can mitigate a number of 
common security vulnerabilities. For example, it used to be common to 
exploit a program by writing code (commonly known as shell code ) to an 
improperly bounds checked string on the stack and changing the saved 
return address of the function to point to the string. By removing the exe¬ 
cute permission from the stack, we can prevent this attack. Most FreeBSD 
architectures do this. 

As expected, breaking simple, stack-based attacks leads attackers to look 
for other vulnerabilities. One of the simplest next steps was to find a way to 
write code to a page that was mapped executable followed by smashing the 
stack to point the return address to it. A popular mitigation for this is the 
write-XOR-execute policy (W A X). This policy prevents mapping pages with 
both the write and execute permission. For most programs, this works with¬ 
out program changes outside the runtime linker, but some programs such as 
Java virtual machines and web browsers use just-in-time (JIT) compilers to 
generate code and run it. These JITs are critical to achieving reasonable per¬ 
formance, but, implemented naively, they don't work with W A X. Fortun¬ 
ately, it is usually a simple matter to map pages writable, write generated 
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code to them, and then make them executable. Not all programs are trivially 
converted though, so W A X implementations generally provide a way to dis¬ 
able W A X for certain programs. 

FreeBSD does not currently support W A X, but work is in progress. The main 
difficulty has been implementing an appropriate framework for tagging bina¬ 
ries that must opt out and providing mechanisms to test opting in or out. We 
have now added a general mechanism (and ELF note) for setting opt-in and 
opt-out bits in binaries as well as flags in procctl which allow features to 
be enabled or disabled in a given execution of a program. We expect to have 
W A X available in FreeBSD 13 and hope to have it enabled by default (at least 
for new programs). The latter part will depend on our confidence in testing 
existing software. 

Setting Maximum Page Permissions 

In the kernel, each page has both a set of permissions and a set of maximum 
permissions. For example, a page backed by a file that the process can read, 
but not write, will not include write permission in either set. Anonymous 
pages have read, write, and execute in all cases in FreeBSD. File-backed pages 
depend on the process's access to the file and any restrictions placed on the 
file descriptor when it was opened. 

While the kernel knows about these, the standard APIs for manipulating 
page permissions— mmap( ) and mprotect( )—don't. This leads to excessive 
default maximum permissions. For example, pages allocated for malloc() 
using mmap( ) default to read and write permission, but can be made exe¬ 
cutable using mprotect( ). In almost no circumstance is that desirable, yet 
today there is no way to prevent it. 

Diversion to CHERI 

Before discussing solutions to this problem, we will consider another problem. 
The CHERI architecture adds a new hardware type, the capability. These capa¬ 
bilities (not to be confused with Capsicum capabilities) grant access to regions 
of virtual memory. They specify a range of accessible addresses along with a 
set of permissions—load, stores, and execute—comparable to the permission 
on pages, but at byte rather than page granularity (typically 4KiB). CHERI 
capabilities are designed to be used at C pointer. In our work on CheriABI, 
we created a process runtime and compilation environment based on 
FreeBSD (our fork is called CheriBSD) where all pointers in a program are 
CHERI capabilities. This includes pointers returned bymmap(). 

When setting bounds on pointers in CheriABI, we attempt to follow the 
principle of least privilege, which states that no more permission should be 
granted than necessary. With pointers returned bymmap( ), our initial inclina¬ 
tion was to return pointers with permissions mapped from the requested 
page protections (e.g., read and write becomes load and store). This mostly 
works but does not work with all usage patterns. First, the runtime linker 
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makes the initial mapping for each library with no permissions, simply reserving 
space. It then maps portions of the file with appropriate permissions. If we 
returned a pointer with no permissions, we would be unable to access the file 
(or make new mappings with more permissions). Second, JITs need to map 
writable to start and then convert those mappings to executable. In both cases, 
we need a way to specify the maximum expected permissions of the mapping 
in order to create a pointer that has those permissions. 

In addressing the problems with mmap( ) in CheriABI, we wanted our 
changes to be minimal. In particular, we wanted the source to continue to work 
on non-CHERI systems and ideally for any mmap( ) extensions to be small 
enough to make it easy to apply them to a cross-platform code base. In the 
end, we decided to steal some bits from the prot argument and add a second 
set of bits for maximum permissions. We created a macro prot_max( ) to be 
ORed with permissions to specify the maximum permissions. For example, a 
library mapping previously mapped with prot_none would be mapped 
PROT_NONE I PROT_MAX(PROT_READ | PROT_WRITE | PROT_EXEC). 
For this code to work on systems that don't support prot_max( ), it can easily 
be defined to 0. To avoid widespread code changes, we chose to treat the sup¬ 
plied permissions as the maximum permissions unless maximum permissions are 
explicitly specified. This required a change to the runtime linker, but, otherwise, 
code in the FreeBSD base system just works with this change in behavior. 

prot_max ( ) in FreeBSD 

When we started exploring W A X, one of the issues that stood out was that vir¬ 
tually all mappings have execute permissions in their maximum permission set. 
We realized that prot_max( ) could address this issue. In June 2019, we com¬ 
mitted a change adding PROT_MAX( ) support to the FreeBSD kernel. It differs 
from the CheriBSD version in that we keep legacy maximum permission logic 
unless the program requests PROT_NONE, a sysctl is set to imply the maxi¬ 
mum permission, or procctl is used to explicitly enable implying maximum per¬ 
missions for this process. 

Much like W A X, we believe that implying maximum permissions with 
prot_max () annotations where required provides the best implementation of 
the principle of least privilege and is the right approach. Work is in progress to 
test the extent to which maximum permission can safely be implied and we 
hope to eventually turn on implying them by default in a future FreeBSD 
release. 

Compatibility and Limitations 

Extensions to inmap ( ) are rarely compatible across platforms. NetBSD has a 
similar extension prot_mprotect ( ) which adds permissions to the maxi¬ 
mum permission set relative to the permission set. This has the disadvantage 
that is doesn't easily allow maximum permissions to be downgraded later via 
mprotect(). 

We found no similar extensions in other operating systems. 
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The prot_max( ) model does have some limitations. You cannot currently use 
it to set a maximum permission of PROT_NONE. You also can't downgrade maxi¬ 
mum permission on a range of pages with mixed permissions without touching 
each sub-range separately or setting all of their permissions identically. 

A major limitation of page-granularity memory permissions is that most pro¬ 
gramming language objects are much smaller than a page. Linkers group togeth¬ 
er objects that should have similar permissions to allow permissions to be limited, 
but it is harder to constrain memory allocated by malloc (). 

Conclusion 

Page-granularity memory permissions are a useful defense against a number of 
attacks. In particular, they allow the application of the principle-of-least-privilege 
to system memory. More advanced systems such as CHERI's capability-based ( 7 ) 

pointers allow further application of fine-grained permissions. 

This article has covered the basics of machine-independent memory permis¬ 
sions that fit the current mmap() permission model. The architecture-specific 
implementations vary and are beyond the scope of this overview as are other 
related mitigations such as Intel's Supervisor Mode Access/Execution Prevention 
(SMAP/SMEP) and ARM's Privilege-Access-Never (PAN), which are related but pro¬ 
tect the kernel from userspace rather than protecting userspace from itself. • 

BROOKS DAVIS has been a FreeBSD developer for nearly two decades and is a member of the FreeBSD 
core team. He has worked on network stacks, high performance computing, build systems, and most recently 
on enhancing FreeBSD for the CHERI architecture. He currently works for SRI International. 
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While there are multiple ways to configure full-disk encryption on 
FreeBSD, this article will focus on one method and provide an easy 
route to follow and get started using GELI. If you already have 
FreeBSD installed on your machine and are looking for instructions 
on how to enable GELI full-disk encryption on a separate disk that 
you attach to your existing install, you'll find the details in the book 
FreeBSD Storage Essentials by Michael W Lucas. 

O n the machine used in this article, we're installing FreeBSD as a fresh 
install and using the normal installer to enable full-disk encryption 
with GELI on ZFS. Go through the installer as you normally would but 
when you get to the Partitioning screen, select Auto (ZFS). Next select 
the Pool Type/Disks option and choose the disk you want to fully encrypt 
and on which you want to install FreeBSD. Choose stripe as the Virtual 
Device type. Select the disk using the Space Bar then press Enter. This will 
bring you back to the ZFS Configuration menu and you can go down to 
Encrypt Disks? and press Enter to change the NO to YES. Go up to Proceed 
with Installation to let the 
utility do the work of 
enabling full-disk encryp¬ 
tion on your fresh 
FreeBSD install. The next 
screen will ask you to 
enter in the passphrase 
that will be used to 
decrypt the disk each 
time you boot the 
machine. We're using a 


Configure Options: 
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Enter a strong passphrase. used to protect gour encryption keys. You mill be required to enter this passphrase each time 
the system is booted 


_ < Cance1> 

Use alpha-numeric, punctuation. TAB or ENTER]- 
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38-character static passphrase on a 
YubiKey along with a passphrase 
that is memorized and manually 
entered prior to the passphrase stored on the YubiKey. First, we enter in the 
memorized passphrase and then press the button on the YubiKey to type out the 
38-character passphrase and press Enter for us. This memorized bit of the 
passphrase prevents a thief from being able to use your stored passphrase from 
your YubiKey to decrypt your machine, assuming they got their hands on both 
your machine and your YubiKey. They would also have to know the first part you 
have memorized, so make sure to keep that a secret. 

You can follow the "Guide to Getting Started with FreeBSD on Virtual and 
Real Hardware" available in the January/February 2019 edition of the FreeBSD 
Journal to get a desktop up and running where you'll be able to interact with 
the YubiKey Personalization utility and program your YubiKey to store a 
passphrase with a maximum of 38 characters. Once you have programmed your 
YubiKey, you can reinstall FreeBSD to get the benefits of full-disk encryption. 
Essentially, the benefits are that if one were to steal your computer or temporari¬ 
ly obtain access to it while it was off, they wouldn't be able to access the files on 
it because the disk is encrypted at rest and the data stored on it isn't accessible 
until GELI is used to decrypt the disk using your passphrase. To program your 
YubiKey, you'll first need to install the software provided by Yubico by typing the 
following command: 


Initializing encryption on selected disks, 
this wi11 take several seconds per disk 


sudo pkg install -y yubikey-personalization-gui 


You can then either use the menu in Lumina to select the newly installed soft¬ 
ware to open it or run the following command to get it to open up: 


yubikey-personalization-gui 


Visit the Static Password tab. Next click on the Scan Code button. The first 
step is to choose the Configuration Slot to use. See the notes provided using the 
? icon next to the Configuration Slot selection bub¬ 
bles for details about the configuration slots and 
instructions on how to use them. Next, in the 
Password section, look for the Keyboard label with 
the Choose-a-Layout—drop-down menu next to 
it—and select the keyboard layout you'll be using. 

Now you can insert your random passphrase into 
the box next to the Password field. Finally, with 
your YubiKey inserted, select Write Configuration 
to save that passphrase to the chosen configura¬ 
tion slot of your YubiKey. Now, when you press 
and hold the button on your YubiKey, you'll see the 


Consoles: EFI console 

6ELI Passphrase for disk0p4: _ 


Sept/Oct 2019 


15 














passphrase automatically type out as if you were to enter it in manually on the 
keyboard. Keep in mind that the amount of time you hold the button varies 
depending on which Configuration Slot you programmed. 

Reboot the machine on which you just installed FreeBSD and the first thing 
you'll see is a screen asking you to enter in your passphrase. Once you have 
entered in your passphrase correctly, the system will boot like normal. • 


Yubico OTP OATH-HOTP Static Password Challenge-Response Settings Too 


Program in Static Password mode - Scan Code 


Select the configure 


ted 





’ubiKey(s) unprotected - Keep it that way 




16 chars for 2.0 and 2.1) 


riiiyL 




Press Write Configuration button to program your YubiKey’s selected configuration slot 


Write Configuration 
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The FreeBSD Journal Editorial Board suggested that some of 
the outstanding interviews from the BSD Now series might 
be of interest to readers as they reflect the state of technolo¬ 
gy when the interview took place. Here, we've transcribed 
and excerpted from a 2014 Interview with Pawel Dawidek 
by Kris Moore and Allan Jude. 
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Kris Moors: BSD Now, Episode 62, Gift from the Sun. We're recording 
October 29, 2014. I'm your host, Kris Moore. 

Allan Jode: And I'm Allan Jude. 

Kris: We're going to be joined by Pawel Dawidek, who has done a lot of 
things in FreeBSD over the years, including the initial ZFS port—and, you 
know, that's kind of a big thing. So, we're going to hear about how that 
came out, what he's up to now, and a whole lot more. Pawel will talk 
about the work he did to port ZFS to FreeBSD. GEOM, GELI, Capsicum— 
various topics. 

Allan : Yeah, it's not just that having ZFS ported to FreeBSD was a huge deal. 
But as he tells the story about how it actually happened, it really gives 
insight into how developing actually gets done. 
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Kris: That would have been a great interview itself but then we get GEOM, 

GELI, Capsicum, and all this other stuff too. 

Allan: So, now we are joined by Pawel Dawidek from The FreeBSD Project. 

Thanks for joining us. 

• Pawel Dawidek: Certainly, thank you. 

Kris: The first question we ask pretty much everybody is what's your backstory, 
how did you first get into BSD? 

• Pawel: / started with computers when I was 12 years old, but at that time I 
was using C-64 and Amiga. And when I went away to study, I used Debian for 
maybe a few months at most. A friend introduced me to FreeBSD and, basically, 

I switched almost directly from Amiga to FreeBSD. One of my problems had 
been that I didn't really have any Windows experience. 

Kris: Is that a problem? [laughs] 

• Pawel: It is a bit of a problem when it comes to family and friends, because if 
you are in IT, they assume you must know this stuff. You must know how to 
install anti-virus programs and things like that. 

Kris: Except that also means you don't get phone calls in the middle of the night 
saying my Windows PC broke again! 

• Pawel: Yes. But I still got those questions. Of course, I loved FreeBSD but 
maybe not from Day 1 because it took me a few tries before I successfully 
installed FreeBSD for the first time. But since then it has been a great adventure. 
Allan: What are you working on currently with FreeBSD? 

• Pawel: I'm not that active anymore, but I still do some things mostly with 
Capsicum and other security projects—my main interest. 

Allan: The other question we traditionally ask is what are some of the things that 
are currently in FreeBSD that we can blame you for? 

• Pawel: Ah yes, probably quite a few of them over the years. I feel like I've 
been a FreeBSD committer for 10 years now. 

Kris: Wow! 

• Pawel: / mostly worked on storage and security, but my code is in many differ¬ 
ent places. In storage, I worked on the various GEOM improvements and GEOM 
classes, like gmirror, gstripe, gconcat, and a few of the other GEOM classes like 
GELI. One of the projects I had the most fun with was porting ZFS to FreeBSD. It 
was a very interesting experience. 

Kris: Since you've brought that up, what was your interest in porting ZFS? What 
brought that on initially? 

• Pawel: / remember the exact moment. During a vacation, a friend told me 
about ZFS. Fie was working on a few things, and they used Solaris a lot. Fie told 
me about the filesystem Sun [Microsystems] was working on, and about features 
like many datasets, end-to-end checksuming, compression, integrated volume 
manager, and it just sounded great. So, I was hoping that when the filesystem 
was created, somebody would port it to FreeBSD. 

But, of course, in FreeBSD, it was a curse with filesystems. We had UFS and 
we've had a lot of failed attempts in porting other filesystems to FreeBSD. 
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I know there was HFS, ReiserFS, read-only. We had XFS, which was also 
read-only. 

Many of them were unfinished, and it was really starting to be a problem. 

So, UFS2 came along, which improved things a bit, but of course some people 
wanted to see ZFS. 

Kris: Sure. 

• Pawel: So, basically, I wanted to see how hard it would be to port ZFS. The 
project was such fun that I got passionate about it. Basically in the first 10 days 
and 10 nights, I was working nonstop. I had so much adrenaline I could actually 
just keep working, and I was sleeping for maybe only 2 hours on some days. 

Kris: Wow. 

• Pawel: And after 10 days, I did have a read-write working prototype. 

It was — 

Kris: the fastest filesystem port ever, [laughs] 

• Pawel: Yes, but of course it wasn't all roses. It took many months—many 
more months actually—to get ZFS into committable shape for FreeBSD. But it 
was great fun. Definitely. 

Allan: What were some of the hurdles you had to overcome to get ZFS into 
FreeBSD and get it all working? 

• Pawel: ZFS was nicely implemented. It was very clean, and it was certainly 
helpful that most of the code could compile in userland. So, the code was pre¬ 
pared to be able to compile in the kernel and in userland—which entailed a lot 
of work to make the code portable. 

I had to create this very ugly layer—a compatibility layer with Solaris, which 
I'm not proud of, but it allowed us to keep the changes to a minimum. The 
difference between us and upstream was extremely helpful when porting new 
versions because, of course, I ported a very early version of ZFS and it was very 
dynamically developed at Sun. So, there were massive changes and this layer 
helped us move forward very quickly. 

ZFS is a filesystem, so it attaches to the VFS layer, which I find one of the 
most complex pieces of FreeBSD's kernel code. Dealing with VFS is tough. 

Kris: I spent a week some months ago looking at Fuse and the VFS layer there. 

• Pawel: I'm a fan of simplifying VFS because our VFS is trying to help filesys¬ 
tem developers by doing a lot of the work for them—centralization and 
buffering and stuff like that. Which, in theory, is nice, but when you deal with 
a more complex filesystem, you do want to have more stuff moved to the 
filesystem itself. 

Allan: Right, you want to have more control as some strategy for buffering is 
different. 

• Pawel: Exactly. ZFS can be much more efficient when it comes to perform¬ 
ance. It has more control over the whole process. 

Allan: Were there any other interesting things that came up when making 
Solaris code work on FreeBSD? 
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• PaWBl: Well, there were some differences, but surprisingly, the kernels' APIs 
were pretty similar. I think there is much less difference between FreeBSD and 
the Solaris kernel than between FreeBSD and Linux kernel. So that was helpful. 
Kris: Makes sense. This question is from our producer, and he's asking what 
would it finally take to get the native ZFS encryption? 

• Pawel: I'm not a fan of how ZFS encryption is implemented in the Oracle 
version. 

Allan: I don't think anybody is. 

• Pawel: Well, some people are. The idea behind Oracle encryption is that you 
encrypt individual datasets. But where do you want to use encryption? Mostly 
on your laptops because you take laptops with you, less so on servers because 
where do you store keys to allow for automatic boot? 

And how many users are on a laptop? Mostly one. And Oracle ZFS encryp¬ 
tion assumes that there are multiple users, and they want to optimize this. So, 
every single dataset can be encrypted with user keys, which is problematic on 
many levels. First of all, it is totally incompatible with the deduplication because 
if you have different datasets using different keys, you have different datasets in 
the pool, so we cannot deduplicate. 

The only way to deduplicate is to clone filesystems so the key is also cloned, 
which is not very nice. 

Allan: Yeah, it is for separate users. 

• Pawel: Yes, exactly. So, I'm not a fan of this idea. I don't think it's really the 
way to go. The way I would implement ZFS encryption is to actually use ZFS 
encryption below the dataset level—encrypt raw data at the vdev level. This way 
we can also encrypt more data because, of course, with Oracle ZFS encryption 
we cannot encrypt zpool-wide metadata. 

Allan: Right. So please talk about the GELI stuff and what you've done with that 
system. 

• Pawel: GELI is basically a disk encryption; it resides below filesystems. There 
are many similar projects. Of course, Linux has its own implementation. GELI is 
obviously a long-running project. 

We did have disk encryption in FreeBSD when I started to work on GELI, but it 
didn't really fit my needs. So, I started to work on my own disk encryption, 
which does provide some unigue features. For example, I don't know of any 
disk encryption software that provides data authentication. 

And GELI was also, from Day 1, integrated with our opencrypto framework, 
so it can use crypto accelerators. GELI also provides many useful features—for 
example, you can use multiple secrets to decrypt your disk, partition, or any 
storage device. You can have a passphrase and a key stored on a pend rive. 

You can also have multiple keys. You can imagine a scenario where in a com¬ 
pany, your security officer has a different key to your master key, and you have 
your own key. So, if your employee loses his key to the data, the data is not 
lost, as there is an alternative key. 

Allan: That's very helpful. 
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Kris: Definitely. Are you still actively involved in GEOM or any filesystem 
development? 

• Pawel: Currently I've mostly moved to security stuff. I do like to observe ZFS 
development these days. I am very happy that there are more and more compa¬ 
nies and people involved because for a few years I think I was the only person 
who was porting ZFS to FreeBSD. And it was definitely not a one-person project. 
Allan: Yes. 

• Pawel: So, / waited for a long time for others to join, and there are quite a lot 
of companies actually using ZFS in production or for their products. So, when 
people are involved in ZFS development on FreeBSD, we have very good relations 
with the illumos community. The TRIM support was one of the developments we 
did in FreeBSD. I did it initially for Multi play from the U.K. And this is something 
that only FreeBSD has as far as I know. There are many stories that companies 
are switching from Linux to FreeBSD to get ZFS. So that's very nice. 

I'm sure many of them were not sure at first but after they switched, I heard 
many positive stories. 

Allan: Would you tell us about your work on Capsicum and what you're doing 
with that now. 

• Pawel: Well, the kernel part of Capsicum is more or less very usable. The thing 
we still work on is Casper daemon. It's a daemon that helps when using 
Capsicum because Capsicum provides a very tight sandbox. You have no access 
to global namespaces. This is actually great technology. 

It's really a great idea and security framework. In the past, I was working on 
my own security stuff—where it was just a bad idea to monitor system calls and 
stuff like that. Capsicum is the way to go, and it gets much more traction. I hope 
that Capsicum will be — 

Kris: the way of the future. 

• Pawel: Yes, and more and more useful. 

Kris: Sure. 

• Pawel: Casper daemon is a way to help develop applications. For example, 
once you are in Capsicum sandbox, you are not able to resolve host names and 
Casper is there to enable you to do that. We are experimenting with Casper as a 
separate daemon, but we are learning it may not be the best idea, as this sepa¬ 
rate process is running in a different context, with its own resource limits, cpuset, 
routing table, etc. We may want to spawn Casper from the application process 
and inherit its context. 

Allan: Yes, so that it doesn't actually provide the easier way around resource limits 
for the CPU site or whatever. 

• Pawel: Exactly. There is some work to do on that front as well. 

Kris: Thanks. What exactly is your day job and do you use FreeBSD there? 

• Pawel: Oh, yes, we do a lot. I'm running my own company, Wheel Systems 
[now Fudo Security]. We are a security vendor; we provide security products. 

It turns out that the technologies I've been working on for FreeBSD over the 
years are very useful in our latest product — Fudo, where we use GELI to encrypt 
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all the storage; we use ZFS to replicate the data between cluster nodes; we use 
compression and snapshots. And we heavily use Capsicum to make it all secure. 

We want to be sure that even if someone breaks into a single session, he can¬ 
not access other sessions. He cannot actually access anything, because if he 
breaks in before authentication, he won't be granted access to connect to the 
server. Only after successful authentication will we provide a connection to the 
destination server. 

And Capsicum makes it really clean and very efficient actually. 

Allan: You don't have to enumerate all the things you can't do. You're saying 
you're only allowed to do these things? 

• Pawel: Yes. This is capability ideology. You only grant the exact rights or access 
to resources that the process requires. Which is not UNIX ideology because, of 
course, if you are running a UNIX program, it has access to everything. 

Allan: Was there anything else you wanted to talk about? 

• Pawel: Not really. • 
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WeGet LETTERS 

W W m by Michael W Lucas 



letters@ 

freebsdjournal.org 


Hey Letters Lackey, 

I keep hearing that security needs to be built in from 
the ground up. I’m stuck using whatever software my compa¬ 
ny says we’re going to use, and a bunch of this software 
stinks. How can I build security from the ground up when I 
have to use cruddy software? I’ve decided the only sensible 
thing to do is stop caring. 


—Indifferent 


Dear Indifferent, 

All three regular readers of this column appear to be drawn by the pleas¬ 
ure of watching my childish behavior when confronted with the tedious duty 
of writing said column. While "you insulted me in the first three words of 
your greeting" is a feeble justification for breaking into your systems and 
converting them to global-warming-accelerating SkunkCoin miners, I'm will¬ 
ing to make it work. 

Because that's what sysadmins do. We make things work. 

Even bad things. 

Because software vendors insist on developing new bad things and cram¬ 
ming them down gullets already obscenely bloated with horrendous bad¬ 
ness. Systems administrators stagger through the endless hours of their brief 
years struggling to live beneath tremendous loads of badness smelted from 
software like arsenic from arsenopyrite. The inherent insecurity of absolutely 
everything enhances this burden like a beached, deceased whale enhances 
an oil spill. 

The urge to retreat into malaise is a natural human reaction. 

Sysadmins lack the luxury of being human. 

The letter writer has already surrendered, so they can stop reading now. 

As this column has three perverted regular readers, however, the editors 
insist I finish this piece with something that resembles useful advice if you 
don't look closely or, indeed, read it. 

So: 

Everything you install is your responsibility. 

It might not be your fault. But it's certainly your responsibility. 

You must be conversant with new software's features. When the 
Tyrannical Paycheck Overlord commands you to install a bucket of sewage, 
you must allocate time to investigate each of the floaty bits in that bucket. 
Making a new service merely run is inadequate; it must run securely. Just as 
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you trawl through a host and exorcise unnecessary daemons, you need to 
sieve those daemons and disable unnecessary features. All of the BSD oper¬ 
ating systems break up monster toolkits like PHP and Perl and even Pascal 
into dozens of individual packages specifically so you can choose to not 
install unnecessary badness. Some other operating systems install the entire¬ 
ty of these toolkits with a single command, giving the inexperienced intrud¬ 
er a banquet of badness to exploit. 

With other horrible software: you don't need a feature? Turn it off. 
Remove unnecessary services, even inside individual packages. It's work. 
You'll inevitably disable features you needed and endure absurd levels of 
hectoring and badgering before you can re-enable them. But the work 
clears your conscience, and when your high-profile organization suffers the 
inevitable mass password snatch you'll be able to tell the charming reporter 
that it was entirely your boss's fault. 

Developers of notably loathsome character produce software where every 
so-called feature is active and cannot be turned off. Programs that purport 
to do everything for everyone. In this worst of all possible worlds, you'll 
inevitably be cornered into deploying and supporting it. How does one 
retain the will to live despite this ineluctable destiny? 

I commend system administration rules seven through ten to your 
attention. 

#7: Temporary solutions aren't. 

Whatever solution you put into place will last far, far longer than you 
intended or hoped. Never slap an ugly hack into place without considering 
its security implications. I've been employed by more than one company that 
had an unsecured modem for emergency access into the network. 
Management knew this modem existed, and specifically described eliminat¬ 
ing it as a goal when I was hired. 

I have implemented redundant VPNs and redundant bandwidth. I have 
integrated authentication systems never intended to interoperate. I have 
restructured entire networks to ensure reliable and secure emergency access 
through any disaster that left the datacenter running. 

I have never been permitted to turn off such an unpassworded emergency 
modem. 

The only solution is to never permit such an abomination on your network 
in the first place. 

#8: Permanent solutions aren't. 

If you stay with an organization long enough, the beautiful new solution 
that solves everything will gradually decay into the stinking albatross around 
the organization's neck. 

I permanently solved my mail problems by building my own mail server 
and installing it at a friend's ISP. The friend moved their office. The hard 
drive failed from old age, so I replaced it. The friend went out of business, 
so I installed a virtual machine. The provider went bust. 
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Everything churns. 

Eventually, that horrible software will churn with it. 

#9: One-off solutions aren't. 

Once you demonstrate that you can solve a problem, people will bring 
other problems to you. For solving, not for laughing derisively at. The obvi¬ 
ous solution might be related to something you whipped up before. At one 
time, I maintained several dozen Perl scripts that differed primarily by the 
degree of stupidity in each. 

Once you demonstrate that you can solve a problem, you officially own 
that entire class of problem. Your Tyrannical Paycheck Overlord gets to 
define the scope of the class. 

Those one-off quick solutions need to be implemented properly, because 
you have to live with them. 

#10: Global solutions aren't. 

Those appalling all-in-one software packages are naturally alluring to 
financial sorts, who think that by buying them, all their problems are solved 
forever. These suites help solve problems, yes—primarily, the problem that 
the software vendor is not yet an oligarch powerful enough to demand the 
excruciation of all who dare question their marketing. Today these systems 
masquerade under names like "Enterprise Resource Planning" or "Customer 
Relationship Management." 

And of course, they're not secure. Because why would they be, when 
running the management interface over telnet is so quick and easy? 

You'll have no choice but to do your best to lock these systems down. 

And you'll wind up implementing a whole bunch of glue to make them 
work as best you can. The best option you have here is to place all the 
blame where it clearly belongs, right on the vendor of this global solution. 

Don't make the mistake of caring for the vendor, though. They're in 
the business of selling solutions, and sysadmin rule #11 is very clear: 
"Solutions aren't." 

Enjoy being doomed. 


Michael W Lucas ( https://m wl.io)'s newest books are Sudo Mastery, 2nd Edition 
and Terrapin Sky Tango. 

If you've read this far, you might 
find FreeBSD Mastery: Jails useful. 

Send your question to 
letters@fre ebs djour n al.com , 

and he might answer it. 

If he can be bothered. 
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conference REP#R1 (s) 


by Benedict Reuschling 

COSCUP ( https://coscup.org/2019/en) is an open-source conference in 
Taipei, Taiwan, that has been running annually since 2006. This year, it 
happened from August 17 and 18, 2019, on the National Taiwan 
University of Science and Technology (NTUST) campus. Since I could 
not attend the BSD TW conference in 2017 (https://2017.bsdtw.org/) 
but people kept telling me what a great time they had there, I decided 
to submit a talk to this year's COSCUP conference. Similar to the FOS- 
DEM BSD devroom, which is a separate track within the main confer¬ 
ence about a specific topic, COSCUP has a BSD track called “BSDTW x 
Cat System Workshop.” Luckily, my talk was accepted, and so I booked 
travel and a hotel for Taiwan. 

I arrived on Wednesday afternoon and after checking into my hotel, I met up with Li-Wen, a 
friend of his, and Hiroki Sato, who had come over from Japan, for dinner. Not only was this an 
excellent opportunity to taste some local dishes, but we could also catch up on current hap¬ 
penings in FreeBSD and plan the next few days together. Li-Wen has been involved in the confer¬ 
ence for a number of years in different capacities, knows a lot of the people as a result, and has 
organized the BSD-specific track there. Additionally, the FreeBSD Foundation had applied for a table 
to talk to people and hand out promotional material like flyers, stickers, and pens. Fortunately, we 
had enough people to man the table, including Philip "trouble" Paeps, Sato-San, Li-Wen, and me. 

Philip had joined us the day before the first official conference day. Fie brought some extra swag 
in a suitcase with which he was travelling around the region from event to event to promote 
FreeBSD. He was also giving a talk about ZFS on Sunday in the BSD track. Together with Hiroki Sato, 
Philip and I went out to dinner together in one of the huge underground metro malls. We talked 
about various things going on in the project and the peculiarities of language learning from differ¬ 
ent perspectives. 

That evening, conference speakers and people staffing the tables were invited to a welcome party 
on a fourth-floor patio overlooking the city, not far from the famous Taipei 101 tower. Similar to the 
BSDCan registration pub, people could pick up their badges the day before the first conference day 
while enjoying a drink and some snacks with other speakers. 

Each person was given their own name badge and it wasn't long until Sato-San and Philip had 
both proper Chinese characters written on theirs by the registration staff. Philip's nickname "trou¬ 
ble" was especially funny to people reading it and proved to be a great conversation starter with 
them. Each new person we met asked him why his name was trouble. Apparently, the characters 
have a double meaning, but in any case, we were not alone for long. 

My own name proved a bit more difficult to transliterate, but the people at the registration desk 
used the phonetic alphabet for mine, and so I also got some nice Chinese characters on my badge. 
After talking to some local people (FreeBSD port maintainers and translators) and meeting Li-Wen 
again (who was busy with some organizational things on the eve of the conference), we left to go 
up the Taipei 101 tower. As the name suggests, this is a 101-story building and the highest in town, 
making for a great landmark. There were long lines waiting for the elevator to take us to the upper 
floor, from which we saw the illuminated city at night. The wind-damper proved also an interesting 
technical aspect as some videos shown there had this massive weight swaying during typhoons and 
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strong winds. 

On the first conference day, I arrived at the NTUST via the local MRT subway network and a 
short walk through a park to the campus. In the weeks prior to the conference, I had received my 
free speaker entrance in the form of a QR-code. As I was about to discover, QR-codes were playing 
a bigger role at the conference than I initially thought. 

I found Li-Wen at the registration desk and after scanning my printed QR-code, I was handed 
another badge and swag bag with sponsor information. Copying my phonetic name to this new 
badge from the one received on the night before proved only partially successful!. The spacing 
between certain characters was a bit off, so I wrote my "western" name below it to avoid confu¬ 
sion. Then I headed for the second floor where Li-Wen had already set up our FreeBSD table. Philip 
was there already, and together, Li-Wen introduced us to the setup he had prepared for visitors. 

A touchscreen was showing two QR-codes that attendees could scan. The first one was a link to 
the BSD Taiwan Facebook group and the second a small questionnaire about FreeBSD that partici¬ 
pants could answer. After they showed us that they completed the survey, we in turn would scan 
their personal participant QR-code and they would get a stamp for visiting our table in the confer¬ 
ence app. With enough stamps collected this way, they could pick up a special prize. Other tables 
around us had similar arrangements from small games to surveys that would keep people visiting 
the table occupied for a few minutes. I found this a great way to engage the attendees more and 
keep them long enough to start a conversation. I've been to other conferences where I felt that 
people just came to the table not because of interest, but to grab free giveaways and then to dis¬ 
appear just as quickly as they came. With the COSCUP stamp game, we got a much better feel for 
the audience, since the survey results showed us that most were still new to FreeBSD. Also, if peo¬ 
ple are shy, we would still get feedback from them. We had some interesting conversations with 
people who had tried out FreeBSD in the past or were running it on their servers for many years. 
Most attendees have not heard or even used FreeBSD before. Clearly, we need to come back. 

What also intrigued me about COSCUP was the high number of staff walking around (all volun¬ 
teers). They were wearing the same t-shirts and had small buttons in their ears with a microphone 
to communicate with each other. For example, if the ice-cold green tea kegs were empty (people 
were thirsty due to the high humidity), they would communicate that and someone came and 
brought a new one. 

The conference day ended at 5 p.m., and we returned to our hotels, but not before visiting a 
local ice cream parlor that had interesting selections like basil or pork knuckle. Sato-San, Philip, 
and I met a couple of hours later for dinner at one of the night markets, which is another must- 
see in Taipei. 

Early the next morning, we got the message that there had been a power break in the confer¬ 
ence building overnight. Since they could not get it fixed on a weekend in time, they moved the 
whole conference (talks, tutorials, tables) to a nearby building on campus that was unaffected by 
the outage and still had power. Another sign of a well-organized conference is probably how well 
the organizers and staff deal with this kind of scenario. The COSCUP folks had not only sent 
emails to all participants with a map of the new location, but they also had found new rooms for 
the talks that day as well as the tables. Signs were posted on campus directing us where to find 
the new building, just as if this had been their plan all along. At no point did I get the impression 
that things were rushed, forgotten, chaotic, or simply not there as a consequence of the power 
outage. A fine job by all the organizers (who probably did not get much sleep the night before). 

While Sato-San stayed at the table and continued scanning visitors' QR-codes, I went to the 
"BSDTW x Cat System" workshop to see Philip's talk about ZFS. It was well attended and I stayed 
on for the next talk about ports by a local FreeBSD maintainer. My own talk was in the afternoon. 
Li-Wen had organized boxed lunches for us for the two conference days, so I was not hungry 
when I gave my talk. The audience was interested in what I had to present, and I got a couple of 
questions from the audience. Some students came to me afterwards and we continued our con¬ 
versation in the hallway. Afterwards, I joined Sato-San at the table. 

My other observation from the conference was that there was a much higher number of 
women attending overall. Some of them had asked us questions at the table or after the talk, and 
from what they asked, I could tell that they were at least on the same technical knowledge level as 
their male peers. This was a very encouraging sight to see. 

Sato-San and I were later interviewed by one of the "BSDTW x Cat System" workshop" group 
members. She also had very precise questions that showed she was not new to the BSDs. Both 
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Sato-San and I gave lengthy answers to her questions about community engagement, how to pro¬ 
mote the BSDs more at conferences like this, and if we had any suggestions for improving COSCUP. 
By the time the interview was over, the last conference talk had also finished and Li-Wen had cleared 
our table and packed away leftover swag. Philip had already left that morning after his talk to catch 
a plane. While walking back to the MRT station, Li-Wen, Sato-San, and I reflected about the confer¬ 
ence and what kind of talks or workshops we should offer there next time. We finished off the day 
with an amazing hot pot dinner at a nearby restaurant. 

Overall, COSCUP proved to be an interesting experience to me. I've been to many conferences in 
different countries over the years. COSCUP left the impression on me that it was professionally 
organized (even without the power outage). There were a lot of helpers (just look at the staff list: 
(http s://coscup.or g /2019/en/staffs ) and many interesting talks, a good amount of which were in 
English. 

I'm glad I went to Taipei (despite the humidity and long travel time). I was greeted very warmly by 
people (sometimes unexpectedly), had a lot of good interactions with the people there, and felt that 
this is definitely a place where we should promote the BSDs more as the crowd seemed interested in 
what the OS can provide.* 


Benedict Beusclling joined the FreeBSD Project in 2009. After receiving his full documentation 
commit bit in 2010, he actively began mentoring other people to become FreeBSD committers. He joined 
the FreeBSD Foundation in 2015, where he is currently serving as vice president. Benedict has a Master 
of Science degree in Computer Science and is teaching a UNIX for software developers class at the 
Darmstadt University of Applied Sciences, Darmstadt, Germany. Together with Allan Jude, he is host of 
the weekly BSDNow.tv ( http://BSDNow.tv ) podcast. 


by Brad Alexander 

Held every other year, I’ve attended the 2015, 2017, and 2019 vBSDcon. 
I work in the Reston, Virginia, area, less than a mile from the venue, the 
Hyatt Regency, and directly across the Dulles Toll Road from confer¬ 
ence-sponsor Verisign, so there wasn’t much travel involved for me 
except for the 50-mile commute home—which did limit some of my 
ability to linger into the night with other attendees. 

still consider myself a relative newbie with FreeBSD, as I have only been using it for approxi¬ 
mately 5 years. A Unix and Linux system administrator for about 25 years, I have touched most 
of the major Unixes. I first delved into FreeBSD thanks to several things falling into place. First, I 
was a regular listener to BSD Now and became, as Kris Moore used to say, "BSD Curious." At the 
same time, Linux began to implement systemd. So, I started looking at FreeBSD because it was about 
as far away from systemd as I could get in an open-source operating system. The final thing, and the 
thing that got me completely hooked, was ZFS, especially ZFS on boot and boot environments. 

My first projects were relatively small. I initially built out a FreeNAS to replace my aging Linux file- 
server that had no drive redundancy. Later, I purchased an HP T610 thin client and installed pfSense 
to replace my Linux firewall. So, my first year, I was totally new to BSD and spent a lot of time talking 
with Kris Moore and Dru Lavigne at the ixSystems and FreeBSD Foundation tables and learning in 
small increments. I joined the ire channels for PC-BSD, TrueOS, FreeNAS. At the time, my impression 
of FreeBSD was that it felt "right down the middle" with respect to the various Unixes I have worked 
with. Each commercial Unix seemed to have its own gimmick that made it feel unique. FreeBSD 
seemed to be one of the purest experiences I have had. 

Each conference that I have attended has taught me more and more, and I have consequently 
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shown up with more confidence each time. The friendliness of the attendees and their willingness 
to suffer newbies was refreshing after many years in the Linux community. This year was no 
exception, and I had a chance to dip my foot further into the world of BSD. The major change for 
me in the past year is that due to the churn in the TrueOS world, I ended up switching my daily 
driver computing platforms, primarily my desktop and laptop, to vanilla FreeBSD 12.0. 

This move allowed me to become more involved with the community. During David Fullard's talk 
on transitioning from FreeNAS to FreeBSD, he mentioned that he wished that freebsd-update would 
be ZFS aware. I had written a script called freebsd-upgrade that did just that. So, I made that avail¬ 
able to him, and we collaborated on the concept. Colin Percival, who wrote freebsd-update had 
some suggestions as well. By the end of the conference, David had a working prototype. 

I also enjoyed Benedict Reuschling's talk on replacing oracle with PostgreSQL, as well as 
Michael W Lucas's talk on jails, which redoubled my intent to transition all my OpenVZ linux con¬ 
tainers to BSD Jails. 

The first night, Benedict educated me on basic Ansible in the hacker lounge, and he showed 
me how to get a basic environment set up. This was one of the times that my commute proved 
an impediment to taking advantage of all I wanted to, both instructional^ and socially. 

Day 2 highlights included Allan Jude's "Explain to Me Like I'm a 5-Year-Old" on ZFS Caching; 
Colin Percival's talk on side channel attacks; and Conor Beh's talk on FreeBSD at work. That hit 
close to home since FreeNAS and pfSense were how I got involved with FreeBSD. I also talked to 
Mark Felder about a bug in pcdm in which services that started after it would not start. He helped 
me to fix it on my system and was looking into a more permanent fix. 

All in all, I thoroughly enjoyed myself at the conference and felt that I came away with deeper 
and wider knowledge, as well as being more involved in the community. My goal is to be able to 
attend BSDCan next year. • 

Brad Alexander After working in satellite communications in the Army, Brad Alexander transitioned 
into IT, where he found a home in Unix system administration. Over the years, he has worked with most of 
the mainstream Unixes, SunOS, Solaris, Xenix, AIX, HP-UX, BSDI, Linux, and FreeBSD. He lives in Virginia 
with his wife of 33 years and their dog. 


♦ EuroBSDcom 2019 at Lillehammer 


by Chris Iroanya 


The four-day EuroBSDcon 2019 was convened in the historic city of 
Lillehammer, Norway, to bring together developers, testers, users, 
and enthusiasts, all for the love of BSD and its variants. I am a network 
administrator, a FreeBSD user, and the recipient of the Paul 
Schenkeveld Travel Grant, which is offered by the EuroBSDcon 
Foundation in honor of his dedication to the BSD community. My home 
country is Nigeria, and the grant supported my travel, hotel accom¬ 
modations, the conference, and the social event. 


L illehammer is rich in history and culture, made clear merely by a walk through the hilly, 
peaceful town. Two Olympics have been hosted here—the 1994 Winter Olympics and the 
2016 Winter Youth Olympics—and a third would have been if Norway had won the bid to 
host the 2022 Winter Olympics. A county administrative center for many years, Lillehammer is also a 
commercially viable tourist destination for lovers of aesthetics and culture as I am. The well-over two- 
hour train ride around the Norwegian countryside was a perfect way for me to see the country's rich 
livestock farming and the beautiful landscape. As a tourist, and a conference attendee, I would say 
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the EuroBSDcon planning committee made a great choice by selecting Lillehammer as our venue. The 
only drawback for me, and I am sure for many other tourists, is the cost of beer—yes, I said "beer." 

The beginning of the first two days of the conference, September 19 and 20, found us either in a 
tutorial or a devsummit. This was a dream come true for me as I am sure it must have been for many 
others, the reason being that I was not only seated in the same room with many of the BSD greats but I 
was also being tutored by them. Another reason was that I was imparted with plenty of useful knowl¬ 
edge and information, with much more to research later. One of my interesting take-homes was from 
Dr. Kirk McKusick's lecture on "An Introduction to the FreeBSD Open-Source Operating System," where 
he explained memory management in the kernel IPC subsystem for the FreeBSD Mbufs and the Routing 
Design using the FreeBSD architecture. 

Day 3 and 4 were even more interesting. I enjoyed the keynote address, "Embedded Ethics," given 
by Patricia Aas. She is by her very nature what I call a "shape shifter," meaning one who is able to 
effect change from her little world. Her talk can be summarized by the Peter Parker principle (Spider- 
Man), "With great power, there must also come great responsibility." Having this in mind, she states 
that we have a moral responsibility to stand up for the ethics of the Information Technology industry, 
despite all odds. 

The cherry on my EuroBSDcon cake was the social event, when we visited the Maihaugen Open Air 
Museum. Spanning a land mass of about 2 square kilometers with 200 houses, it is the most visited 
tourist attraction in Lillehammer, and it was great that we added to the count. The museum prides itself 
on the preservation of Norwegian culture and history, which it represents through the three primary sec¬ 
tions in the museum: the Rural Collection; Historic Town; and the Residential Area. This is indeed a 
must-see for anyone visiting Norway. 

The conference closed at about 3:30 p.m. on the 22nd with the declaration that the next 
EuroBSDcon will be hosted in Vienna, Austria. We all greeted it with a loud cheer, knowing very well of 
Austria's richly brewed and affordable beer. 

In summary, my experience at the EuroBSDcon 2019 was wonderful. It has given me a lot to research 
and provided possible project ideas to explore. Thank you to the FreeBSD Foundation, the FreeBSD com¬ 
munity, the conference organizers, the speakers, and everyone who has kept this project alive. Looking 
forward to seeing you again in Vienna, Austria. Until then, "Habe ein tolles BSD-Entwicklungsjahr." 

Hope I got that right and that it means, "Have a great BSD development year" in German. • 

Chris Iroanya Chris Iroanya is a network administrator with Wilforce International Services Limited, 

and sets up network servers using the FreeBSD OS. He lives in Nigeria with his wife, Mimi, and two 

lovely girls, Samantha and Chimamanda. He loves cooking and spending time with his family. 


Contact walter@fbsdjournaI.com 
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by anne dickison The Following BSD Events will 
take place through February 2020 


linux.conf.au 2020 • January 13-17, 2020 • Gold Coast, Australia 

https://linux.conf.au/programme/miniconfs/freebsd/ 


The FreeBSD Foundation, in conjunction with simPRO Software, 
is excited to host a FreeBSD Miniconf on January 14, 2020, at linux.conf.au. The call for sessions is open 
until November 17, 2019. 


^r 


FreeBSD Mini Developers Summit • January 31, 2020 

* Brussels, Belgium https://wiki.freebsd.org/Devsummit /202002 

Join fellow FreeBSD developers for a one day event co-located with FOSDEM '20. 
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FOSDEM 2020 • Feb 1 & 2 • Brussels, Belgium 

https://fosdem.org/2020/ Taking place February 1 &2, 2020 in Brussels, Belgium, 
FOSDEM offers open-source and free software developers a place to meet, share 
ideas, and collaborate. Renowned for being highly developer-oriented, the event 
brings together some 8,000+ geeks from all over the world. 


^T 



Have Good Ideas? 

Write 
for Us! 


Contact Jim Maurer, Editor-at-Large. 

jmaurer@freebsdjournal.com 
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